PO04-2006: PUB_BIAS: A SAS® Macro for Detecting Publication Bias in Meta-Analysis

Publication bias is one threat to validity that researchers conducting meta-analysis studies confront. Although statistical methods for detecting publication bias have surfaced in the literature (e.g., Begg Rank Correlation, Egger Regression, Funnel Plot Regression, and Trim & Fill), many researchers rely on visual inspection of funnel plots. This program was created to provide meta-analysts with the ability to implement statistical methods to detect publication bias. Statistical information from the corpus of studies (i.e., effect sizes and sample sizes) is input in this SAS macro and analyzed for potential publication bias using the above mentioned statistical methods. In addition to the p-value associated with each detection method the program output provides the estimated mean effect size and random effects variance component. The application of the SAS/IML programming on two sets of example data (with and without publication bias) is provided along with the macro programming language. INTRODUCTION Publication bias is an important issue that researchers face when conducting a literature review, designing a new study, or conducting a meta-analysis. Unfortunately, when a researcher gathers literature their findings do not include all studies that have occurred regarding the specified content area searched. This phenomenon was discussed by Rosenthal (1979) as the “file drawer problem” or publication bias. Essentially, researchers may have studies that are sitting in their filing cabinets because they decided not to publish or were rejected by journals. Reasons researchers do not submit studies or for journals to reject studies typically revolve around whether the results indicated significant findings, which are influenced by sample size, or large effects. In addition, published research can inadvertently contribute to publication bias when researchers exclude non-significant findings from results or report data poorly. Thus, there is a pattern in the published literature of a disproportionate number of studies with statistically significant findings and large effects. When meta-analysts do not include unpublished studies, the results of the meta-analysis may be biased. Specifically, the meta-analysis results may indicate an inflated effect because the published studies are more likely to have significant results and large effects (Sharpe, 1997). Thus, publication bias is considered to be a threat to the validity of meta-analyses. One method for detecting publication bias is the visual interpretation of a funnel plot (a scatterplot of effect sizes and sample sizes). However, visual examination of the funnel plot is limited because the interpretation is subjective and the plot can be difficult to interpret when there are a small number of studies included in the metaanalysis (Greenhouse & Iyengar, 1994; Thornton & Lee, 2000). Consequently, some researchers have developed statistical methods for detecting publication bias that are not subjective. IMPACT OF PUBLICATION BIAS ON META-ANALYTIC SUMMARIES Using a Monte Carlo design Rendina-Gobioff (2006) examined the impact of moderate and strong publication bias on the estimated mean effect size and the estimated effect size variance results of random-effects meta-analyses. Consistent with the literature Rendina-Gobioff (2006) found that when no publication bias was imposed the average effect size bias was -0.0120 with a minimum value of -0.1125 and maximum value of 0.0997. In contrast, the average effect size bias increased to 0.0792 when moderate publication bias was imposed, with a minimum value of -0.0327 and maximum value of 0.3030. The average effect size bias increased even more when the imposed publication bias was strong, 0.1350 (minimum= -0.0295 and maximum=0.4491). According to these results when a researcher is conducting a meta-analysis with strong publication bias they could be producing an average effect size with as much as 0.45 error. Similar to bias associated with the mean effect size estimates, when there is no publication bias one would expect to have minimal effect size variance bias. Consistent with this assumption Rendina-Gobioff (2006) found that when no publication bias was imposed the average effect size variance bias was -0.0343 with a minimum value of -0.3806 and maximum value of 0.2756. In contrast, the average effect size variance bias increased to 0.1101 when moderate publication bias was imposed, with a minimum value of -0.1593 and maximum value of 0.7757. The average effect size variance bias increased even more when the imposed publication bias was strong, 0.2052 (minimum=-0.1413 and maximum=1.1622). According to these results when researchers are conducting a meta-analysis with strong publication bias they could be producing an average effect size variance estimate with as much as 1.16 error.